1. Field of the Invention
The present invention relates to an image processing apparatus and method, and to a medium therefor and, more particularly, relates to an image processing apparatus and method for measuring a distance to an object by using a plurality of stereo image pairs, and to a medium therefor.
2. Description of the Related Art
As shown in, for example, FIG. 1, a stereo image distance measurement system is known in which the same object 5 is photographed by a standard camera 1 and a reference camera 2, and in an image processing apparatus 31, corresponding pixels of a pair of photographed images (a standard image and a reference image A) are specified, and a distance from the standard camera 1 to the object 5 is determined by using parallax information.
The operation of this stereo image distance measurement system is described below. FIG. 2 shows the relationship between a reference image A and a standard image in which an object point P in a three-dimensional space is photographed by the standard camera 1 or the reference camera 2. That is, if a pixel corresponding to the object point P within the standard image is denoted as a pixel mb and a pixel corresponding to the object point P within the reference image A is denoted as a pixel mi, the coordinates of the pixel mi are as described below by using a projection transformation matrix H of 3 rows and 3 columns:
mi=Hxc2x7mb
Therefore, the projection transformation matrix H can be computed if the coordinates of the pixels mb and mi are determined.
Accordingly, in a system in which the positional relationship between the standard camera 1 and the reference camera 2, and a distance Z from the standard camera 1 to the object point P are measured in advance, the coordinates of the pixels mb and mi are specified, and the projection transformation matrix H corresponding to the distance Z is computed in advance by using these coordinates. The computed projection transformation matrix H is stored in the image processing apparatus 31.
As shown in, for example, FIG. 1, when the distance from the standard camera 1 to the object point P is to be determined, the image processing apparatus 31, as shown in FIG. 3, assumes the distance to be Z1 from among the distances Z1 to Zn which are discretely set in a distance measurement range, and determines a reference image mi1 by using the projection transformation matrix H, corresponding to the distance Z1, in which the corresponding pixel (standard pixel) mb on the standard image is computed in advance and stored such that the object point P is photographed by the standard camera 1.
Thereafter, the image processing apparatus 31 computes the mutual correlation value S1 between the determined reference image mi1 and the standard pixel mb. The mutual correlation value S1 is computed by using the normalized mutual correlation between a predetermined small area with the pixel mb on the standard image being the center and a similar small area with the pixel mi1 in the reference image A being the center, and an evaluation function, such as the sum of the squares of a luminance difference.
Next, assuming the distance from the standard camera 1 to the object point P to be Z2, the image processing apparatus 31 determines a reference pixel mi2 by using a projection transformation matrix H corresponding to the distance Z2 which is computed in advance and stored. Thereafter, the image processing apparatus 31 computes a mutual correlation value S2 between the determined pixel mi2 and the pixel mb.
Hereafter, in a similar manner, assuming a distance from the standard camera 1 to the object point P to be Z3 to Zn in sequence, the image processing apparatus 31 determines the reference pixels mi3 to min by using a projection transformation matrix corresponding to the assumed distance and computes the mutual correlation values S3 to Sn.
It is known that pixels mil to min, in which the pixel mb on the standard image is projected within the reference image A, are present on one straight line called an xe2x80x9cepipolar linexe2x80x9d, as shown in FIG. 3.
FIG. 4 shows the relationship between the distances Z1 to Zn and the mutual correlation values S1 to Sn. In the figure, it is shown that the smaller the value S of the mutual correlation value, the higher the correlation between pixels (the compared pixels are more similar).
Next, the image processing apparatus 31 creates a curve (for example, a second-order curve) passing through the smallest (highest correlation) one (a mutual correlation value S3 in the example of FIG. 4) from among the mutual correlation values S1 to Sn and the values (the mutual correlation values S2 and S4 in the example of FIG. 4) before and after it, and outputs a distance Ze corresponding to the smallest value Se thereof as a distance of the standard pixel mb.
FIG. 5 shows the construction of a stereo image distance measurement system which measures the distance from the standard camera 1 to the object 5 by using two sets of stereo image pairs (a standard image and a reference image A, and a standard image and a reference image B). For the operation thereof, a process similar to the construction shown in FIG. 1 is performed on two sets of stereo image pairs. That is, as shown in FIG. 6, an image processing apparatus 41 determines the pixels mi1 to min in the reference image A corresponding to the pixel mb on the standard image and computes the mutual correlation values S1 to Sn with respect to the pixel mb. Also, the image processing apparatus 41 determines pixels mi1 to Min on the reference image B corresponding to the pixel mb in the standard image, and computes the mutual correlation values S1xe2x80x2 to Snxe2x80x2 with respect to the pixel mb.
Thereafter, the image processing apparatus 41 adds together the mutual correlation values S1 to Sn and the corresponding mutual correlation values S1xe2x80x2 to Snxe2x80x2. FIG. 7 shows that the mutual correlation values S1 to Sn are indicated by xe2x97xaf marks, the mutual correlation values S1 to Snxe2x80x2 are indicated by ▴ marks, and the mutual correlation values SS1 to SSn are indicated by xe2x96xa1 marks.
Next, the image processing apparatus 41 selects the smallest (highest correlation) one (the addition value SS4 in the example of FIG. 7) from among the addition values SS1 to SSn, and creates a curve (for example, a second-order curve) passing through the selected value and the values (the addition values SS3 and SS5 in the example of FIG. 7) before and after it, and outputs a distance Ze corresponding to the smallest value SSe as a distance of the standard pixel mb.
In a manner as described above, use of two sets of stereo image pairs makes it possible to obtain a result with higher accuracy than when one set of stereo image pairs is used.
In the above-described stereo image distance measurement system, in order to determine the pixel on the standard image corresponding to the pixel mb on the standard image plane, a search is made (mutual correlation values are compared) discretely on the epipolar line. In order to measure a distance with a higher accuracy, it is preferable that a search be made on the epipolar line at shorter intervals.
However, on the epipolar line, if the number of searches is increased, the number of computations is increased correspondingly, resulting in problems in that the scale of the system is increased and the processing time is increased.
The present invention has been achieved in view of such circumstances. An object of the present invention is to make it possible to measure a distance with a higher accuracy without increasing the number of searches which must be performed.
To achieve the above-mentioned object, according to one aspect of the present invention, there is provided an image processing apparatus for measuring a distance to an object by using a plurality of images of the same photographed object, the image processing apparatus comprising: first extraction means for extracting, from a first reference image, a first candidate pixel which is a candidate for a first corresponding pixel corresponding to a desired pixel on a standard image, for each distance in a first group of distances formed of a plurality of distances which are set discretely in a measurement range; second extraction means for extracting, from a second reference image, a second candidate pixel which is a candidate for a second corresponding pixel corresponding to a desired pixel on the standard image, for each distance in a second group of distances formed of a plurality of distances, different from those of the first group of distances, which are set discretely in a measurement range; computation means for measuring a similarity between the desired pixel on the standard image and the first candidate pixel on the first reference image, and a similarity between the desired pixel on the standard image and the second candidate pixel on the second reference image; and determination means for determining the distance to the object on the basis of the result computed by the computation means.
According to another aspect of the present invention, there is provided an image processing method for measuring a distance to an object by using a plurality of images of the same photographed object, the image processing method comprising: extracting, from a first reference image, a first candidate pixel which is a candidate for a first corresponding pixel corresponding to a desired pixel on a standard image, for each distance in a first group of distances formed of a plurality of distances which are set discretely in a measurement range; extracting, from a second reference image, a second candidate pixel which is a candidate for a second corresponding pixel corresponding to a desired pixel on the standard image, for each distance in a second group of distances formed of a plurality of distances, different from those of the first group of distances, which are set discretely in a measurement range; measuring a similarity between the desired pixel on the standard image and the first candidate pixel on the first reference image, and a similarity between the desired pixel on the standard image and the second candidate pixel on the second reference image; and determining the distance to the object on the basis of the result computed by the computation means.
According to yet another aspect of the present invention, there is provided a medium for providing instructions to an image processing apparatus to execute a process for measuring a distance to an object by using a plurality of images of the same photographed object, the process comprising the steps of: extracting, from a first reference image, a first candidate pixel which is a candidate for a first corresponding pixel corresponding to a desired pixel on a standard image, for each distance in a first group of distances formed of a plurality of distances which are set discretely in a measurement range; extracting, from a second reference image, a second candidate pixel which is a candidate for a second corresponding pixel corresponding to a desired pixel on the standard image, for each distance in a second group of distances formed of a plurality of distances, different from those of the first group of distances, which are set discretely in a measurement range; measuring a similarity between the desired pixel on the standard image and the first candidate pixel on the first reference image, and a similarity between the desired pixel on the standard image and the second candidate pixel on the second reference image; and determining the distance to the object on the basis of the result computed by the computation means.
In the image processing apparatus in accordance with the present invention, first extraction means extracts, from a first reference image, a first candidate pixel which is a candidate for a first corresponding pixel corresponding to a desired pixel on a standard image, for each distance in a first group of distances formed of a plurality of distances which are set discretely in a measurement range; second extraction means extracts, from a second reference image, a second candidate pixel which is a candidate for a second corresponding pixel corresponding to a desired pixel on the standard image, for each distance in a second group of distances formed of a plurality of distances, different from those of the first group of distances, which are set discretely in a measurement range; computation means measures a similarity between the desired pixel on the standard image and the first candidate pixel on the first reference image, and a similarity between the desired pixel on the standard image and the second candidate pixel on the second reference image; and determination means determines the distance to the object on the basis of the result computed by the computation means.
In the image processing method and the medium therefor in accordance with the present invention, in the first extraction step, a first candidate pixel which is a candidate for a first corresponding pixel corresponding to a desired pixel on a standard image is extracted from a first reference image, for each distance in a first group of distances formed of a plurality of distances which are set discretely in a measurement range. In the second extraction step, a second candidate pixel which is a candidate for a second corresponding pixel corresponding to a desired pixel on the standard image is extracted from a second reference image, for each distance in a second group of distances formed of a plurality of distances, different from those of the first group of distances, which are set discretely in a measurement range. In the measurement step, a similarity is measured between the desired pixel on the standard image and the first candidate pixel on the first reference image, and a similarity is measured between the desired pixel on the standard image and the second candidate pixel on the second reference image. In the determination step, the distance to the object is determined on the basis of the result computed by the computation means.
The above and further objects, aspects and novel features of the invention will become more apparent from the following detailed description when read in connection with the accompanying drawings.